Playing with the Loki Stack

Let’s get our hands dirty. The first step is to set up a cluster and install the Loki stack and a few other tools.

Get started with Loki#

We made all the preparations, we deployed all the apps, and we applied all the configuration we need to give Loki a spin. Now we're ready to dive in.

We'll open Grafana first since that will be our user interface.

Note: The nip.io URL works with local addresses. That’s good news because that means it can work with Docker Desktop and Minikube. The bad news is that we might have told you otherwise in the previous chapters. We should have known that it's improved since the last time we experimented with it.

We're presented with the login screen. The username is "admin," but the default password is not a hard-coded value like "admin" or "123456." That’s a good thing in general, but it also means that we need to retrieve it.

The password is stored in the field admin-password inside the Secret grafana. All we have to do is retrieve it and decode it.

Copy the password and go back to the login screen. Type "admin" as the username and paste the password.

The next step is to click the "Log in" button.

We're presented with the input line (the one next to the drop-down with "Log labels" selected). That’s where we can type queries, so let’s try something simple.

We're asking it to retrieve all the logs belonging to the job called production/go-demo-9-go-demo-9. The job is one of the labels automatically injected by Promtail. We can think of it as the Deployment (go-demo-9-go-demo-9) with a prefix containing the namespace (production). The two combined provide a unique ID since there cannot be two Deployments in the same namespace with the same name.

Press the "Shift" and "Enter" keys to execute the query. Or click the "Run Query" button in the top-right corner of the screen.

Note: We'll skip reminding you to run queries. Whenever we instruct you to type or change a query, we'll assume that you'll execute it.

View the logs#

We should see all the logs coming from go-demo-9-go-demo-9 running in the namespace production. Or, to be more precise, we won’t see all the logs, but rather those generated during the last hour. The period of the logs retrieved can be adjusted through the drop-down in the top menu. It's currently set to the "Last 1 hour."

Feel free to play with the drop-down lists and the buttons. Explore it for a while. For example, you could expand one of the log entries to see the additional information.

The output contains logs of our application, but we're swarmed with GET request to / lines, so it's hard to see whatever we might want to see.

Let's modify the query with the snippet that follows.

Voila! The annoying noise is gone. The output is limited to all the log entries that do NOT contain GET request to /. We told Loki that we want the log entries with values with labels job being production/go-demo-9-go-demo-9 but limited to those that are different than GET request to /. The != command means "different than."

Note: You might not see any logs in your application. If that’s the case, you may be too slow. Those matching the query were generated over an hour ago. Please increase the duration if that’s the case.

If we're familiar with Linux and want to accomplish a similar outcome based on logs stored in a file, we would probably execute a command similar to the one that follows.

Note: Do not run the command that follows. It's meant to show the equivalent “old” way of dealing with logs. You might not have the log file it uses, so the command would fail anyway.

This is precisely what we’ve got in Loki, but in a dynamic environment of a Kubernetes cluster with possibly hundreds of replicas and dozens of release versions of our application. We can think of Loki as a solution for “distributed grep” commands.

Now that we have a taste of what Loki does, let’s step back and get a more formal introduction into its syntax and capabilities.

Note: We'll get hands-on experience with the concepts and commands discussed in this lesson in the project "Hands-on: Using Centralized Logging" right after this chapter.